Replace generated byte array with go:embed for templates by cardil · Pull Request #3597 · knative/func

cardil · 2026-04-09T12:04:48Z

Summary

Replaces the codegen-based template embedding with direct //go:embed templates/, eliminating the 14K-line generated file generate/zz_filesystem_generated.go and the codegen tool that produced it.

Problem

The old approach used generate/templates/main.go to walk the templates/ directory, zip its contents in memory, and write the result as a Go byte array literal (var TemplatesZip) into generate/zz_filesystem_generated.go (~14K lines of hex). This required running codegen before any build or test, produced a large committed generated file, and made it harder to inspect embedded template contents or review template diffs.

Solution

Use //go:embed directly on the templates/ directory tree, working around two hard go:embed limitations:

Directories with go.mod are silently skipped → rename go.mod/go.sum to go.mod.embd/go.sum.embd
Symlinks are silently dropped → replace symlinks with .symlink plain-text marker files containing the target path

A new manglingFS wrapper in pkg/filesystem transparently reverses this mangling at runtime:

*.embd files are exposed with their real names
*.symlink files are presented as symlink directory entries
//go:build embd constraints are stripped from Go source file content
File permissions (lost by embed.FS which returns 0444 for everything) are restored from a committed templates/.permissions manifest

New Tools

hack/cmd/embd — embd/unembd subcommands to temporarily unmangle a template directory so normal Go tooling (go mod tidy, go get, go test) can operate on it, then re-mangle it. Uses the same constants/logic as manglingFS to keep mangling rules in sync.
hack/cmd/permissiongen — codegen tool that scans templates/ and generates templates/.permissions listing non-standard file permissions (only executable files differ from the default 0644). Run via go run ./hack/cmd/permissiongen.

Build tag approach

Go template source files are tagged with //go:build embd so the Go toolchain does not attempt to compile them as part of the main module. The go/build/constraint package is used for correct AST-based manipulation of build tags (preserving any existing tags on round-trips through embd/unembd).

Changes

templates/embed.go — //go:embed directive
pkg/filesystem/mangling.go — manglingFS runtime FS wrapper
pkg/functions/templates_embedded.go — wires up manglingFS
hack/cmd/embd/main.go — embd/unembd CLI tool
hack/cmd/permissiongen/main.go — permissions manifest codegen
templates/.permissions — committed generated manifest
Makefile — updated check-go, test-go and update-runtime-go targets
hack/update-codegen.sh / hack/verify-codegen.sh — updated
.gotagsignore — excludes embd from CI build tag matrix
Removed: generate/zz_filesystem_generated.go, generate/templates/main.go

Assisted-by: 🤖 Claude Opus/Sonnet 4.6

- generate/templates/main.go: write binary templates.zip instead of Go source - generate/templates/gen.go: extract zip-writing logic (with .gitignore filtering) - generate/templates/check.go: 3-tier verification (exist/stale/hash) - generate/embed.go: //go:embed templates.zip exposes TemplatesZip []byte - delete generate/zz_filesystem_generated.go - .gitignore: add generate/templates.zip - .gitattributes: remove zz_filesystem_generated.go entry - Makefile: update targets/clean to use generate/templates.zip - hack/verify-codegen.sh: use Go checker for templates.zip verification - pkg/functions/templates_embedded.go: run whole package instead of single file - pkg/filesystem/filesystem_test.go: update stale error message Assisted-by: 🤖 Claude Sonnet 4.6

knative-prow · 2026-04-09T12:05:00Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: cardil
Once this PR has been reviewed and has the lgtm label, please assign jrangelramos for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

matejvasek · 2026-04-09T12:36:25Z

Would this work even if I imported function as a library (We doing that now in Functions OCP dynamic plugin)?

lkingland · 2026-04-10T10:51:52Z

Sorry, but we tried go:embed as soon as it was available, but it doesn't work for our use-case for multiple reasons.

EDIT: Ah, I see we have a Slack thread about this.

**The generated Go source file changed in almost every PR that touched templates, causing constant merge conflicts. A binary zip + //go:embed eliminates this entirely since templates.zip is gitignored.

tl;dr, This could perhaps be minimized by splitting the monolithic generated file into a per-language, per-template set of generated files.

cardil · 2026-04-10T11:05:52Z

I think I have a solution. Will keep you posted...

/hold

- Rename go.mod/go.sum → .embd suffix (go:embed skips dirs with go.mod) - Convert symlinks → .symlink marker files (go:embed drops symlinks) - Add //go:build embd to Go template source files - Add templates/embed.go with //go:embed directive - Add pkg/filesystem/mangling.go: manglingFS transparently reverses mangling at runtime (strips build tags, unmangles names, restores perms) - Add hack/cmd/permissiongen: codegen for templates/.permissions manifest (preserves file permissions lost by embed.FS which returns 0444 for all) - Add hack/cmd/embd: embd/unembd subcommands for Go tooling compatibility (test-go, update-runtime-go targets need real go.mod to work) - Update Makefile: use hack/cmd/embd for test-go and update-runtime-go - Update hack/update-codegen.sh and hack/verify-codegen.sh - Remove generate/ zip infrastructure (embed.go, templates/gen.go etc) - Add .gotagsignore to exclude embd tag from CI build matrix - Export EmbdSuffix, SymlinkSuffix, EmbdBuildConstraint, StripEmbdBuildTag, AddEmbdBuildTag from pkg/filesystem for reuse by hack/cmd/embd - Use go/build/constraint for proper AST-based build tag manipulation Assisted-by: 🤖 Claude Sonnet 4.6

Assisted-by: 🤖 Claude Sonnet 4.6

…emplates

cardil · 2026-04-17T18:06:49Z

/unhold

- check-go: use hack/cmd/embd unembd/embd around go vet/lint - Remove make generate/zz_filesystem_generated.go from: update-python-platform.yaml, update-ca-bundle.js, update-quarkus-platform.js, update-springboot-platform.js Assisted-by: 🤖 Claude Opus 4.6

…e-go Scaffolding symlinks (f → ../../http/) cross directory boundaries, so individual subdir unembd leaves targets in mangled state. Unmangle the whole templates/go tree at once instead.

codecov · 2026-04-17T18:42:54Z

Codecov Report

❌ Patch coverage is 59.93151% with 117 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.36%. Comparing base (02faab0) to head (5a965bb).

Files with missing lines	Patch %	Lines
pkg/filesystem/mangling.go	59.93%	104 Missing and 13 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3597      +/-   ##
==========================================
+ Coverage   56.30%   56.36%   +0.05%     
==========================================
  Files         180      180              
  Lines       20549    20836     +287     
==========================================
+ Hits        11570    11744     +174     
- Misses       7778     7879     +101     
- Partials     1201     1213      +12

Flag	Coverage Δ
e2e	`36.39% <55.47%> (+0.16%)`	⬆️
e2e go	`33.11% <58.10%> (+0.24%)`	⬆️
e2e node	`28.68% <40.71%> (+0.06%)`	⬆️
e2e python	`33.26% <45.84%> (+0.03%)`	⬆️
e2e quarkus	`28.83% <41.50%> (+0.07%)`	⬆️
e2e rust	`28.25% <40.71%> (+0.13%)`	⬆️
e2e springboot	`26.75% <41.50%> (+0.10%)`	⬆️
e2e typescript	`28.81% <40.71%> (+0.07%)`	⬆️
e2e-config-ci	`18.54% <53.75%> (+0.46%)`	⬆️
integration	`17.39% <12.64%> (-0.09%)`	⬇️
unit macos-14	`43.60% <58.49%> (+0.25%)`	⬆️
unit macos-latest	`43.60% <58.49%> (+0.24%)`	⬆️
unit ubuntu-24.04-arm	`43.74% <56.50%> (+0.19%)`	⬆️
unit ubuntu-latest	`44.46% <58.49%> (+0.23%)`	⬆️
unit windows-latest	`43.61% <58.49%> (+0.23%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Template Go files are now part of the root module (go.mod renamed to go.mod.embd), so go mod tidy resolves their imports.

On Windows, FilesystemFromRepo returns nil for file:// URIs. Guard against nil before passing to NewManglingFilesystem to avoid nil pointer dereference in loadPerms.

cardil · 2026-04-18T08:44:38Z

E2E Runtime failures (go, python, quarkus, rust, typescript) are infra flakes: Docker Hub rate limiting (TOOMANYREQUESTS) and job timeouts. Go E2E TestMatrix_Run and TestMatrix_Deploy all passed — only TestMatrix_Remote was cancelled mid-run.

/retest

cardil · 2026-04-18T09:26:29Z

Thanks for the feedback @matejvasek @lkingland. Your comments were about the earlier version of this PR, which took a different approach. The PR has been fully reworked since then.

@matejvasek good point about library import. The current approach uses //go:embed templates/ directly in the templates/ package. Go resolves embed directives at build time relative to the source package, so the embedded templates travel with the module when imported as a dependency. The public API (Filesystem interface from pkg/functions) hasn't changed. The manglingFS wrapper that reverses the file mangling is transparent to callers.

@lkingland you're right that raw go:embed doesn't work for our templates because dirs with go.mod get silently skipped and symlinks are dropped. This version works around both:

go.mod/go.sum are renamed to go.mod.embd/go.sum.embd at rest
Symlinks are replaced with .symlink marker files containing the target path
Go source files in templates get a //go:build embd tag so the toolchain won't try to compile them

A manglingFS wrapper reverses all of this at runtime. Consumers see real go.mod files, real symlinks, and untagged Go source.

The main motivation was merge conflicts. The old zz_filesystem_generated.go (14K lines of hex byte array) changed in almost every template PR. With direct go:embed, the template files are committed as-is (with mangling), so diffs are human-readable and conflicts are trivially resolvable. No generated file to maintain.

The tradeoff is the mangling tooling (hack/cmd/embd for embd/unembd, hack/cmd/permissiongen for the permissions manifest), but these are small, well-tested tools that keep the mangling rules in sync with the runtime manglingFS.

knative-prow bot requested review from dsimansk and jrangelramos April 9, 2026 12:04

knative-prow bot added the size/L 🤖 PR changes 100-499 lines, ignoring generated files. label Apr 9, 2026

matejvasek requested a review from lkingland April 9, 2026 12:27

knative-prow bot added the do-not-merge/hold 🤖 PR should not merge because someone has issued a /hold command. label Apr 10, 2026

cardil marked this pull request as draft April 10, 2026 11:05

knative-prow bot added the do-not-merge/work-in-progress 🤖 PR should not merge because it is a work in progress. label Apr 10, 2026

knative-prow-robot added the needs-rebase Cannot be merged due to conflicts with HEAD. label Apr 17, 2026

knative-prow bot added size/XXL 🤖 PR changes 1000+ lines, ignoring generated files. and removed size/L 🤖 PR changes 100-499 lines, ignoring generated files. labels Apr 17, 2026

cardil added 2 commits April 17, 2026 19:59

Remove stale /generate/templates.zip from .gitignore

2af95c3

Assisted-by: 🤖 Claude Sonnet 4.6

Merge remote-tracking branch 'upstream/main' into refactor/go-embed-t…

c46327e

…emplates

knative-prow-robot removed the needs-rebase Cannot be merged due to conflicts with HEAD. label Apr 17, 2026

cardil marked this pull request as ready for review April 17, 2026 18:06

knative-prow bot removed the do-not-merge/work-in-progress 🤖 PR should not merge because it is a work in progress. label Apr 17, 2026

knative-prow bot removed the do-not-merge/hold 🤖 PR should not merge because someone has issued a /hold command. label Apr 17, 2026

cardil added 2 commits April 17, 2026 20:15

fix: unmangle entire templates/go tree for check-go and update-runtim…

313d672

…e-go Scaffolding symlinks (f → ../../http/) cross directory boundaries, so individual subdir unembd leaves targets in mangled state. Unmangle the whole templates/go tree at once instead.

cardil added 2 commits April 17, 2026 20:44

fix: go mod tidy to pick up embedded template deps

0961ccc

Template Go files are now part of the root module (go.mod renamed to go.mod.embd), so go mod tidy resolves their imports.

fix: guard nil FS in initGitFS before wrapping with ManglingFilesystem

5a965bb

On Windows, FilesystemFromRepo returns nil for file:// URIs. Guard against nil before passing to NewManglingFilesystem to avoid nil pointer dereference in loadPerms.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace generated byte array with go:embed for templates#3597

Replace generated byte array with go:embed for templates#3597
cardil wants to merge 8 commits intoknative:mainfrom
cardil:refactor/go-embed-templates

cardil commented Apr 9, 2026 •

edited

Loading

Uh oh!

knative-prow bot commented Apr 9, 2026

Uh oh!

matejvasek commented Apr 9, 2026

Uh oh!

lkingland commented Apr 10, 2026 •

edited

Loading

Uh oh!

cardil commented Apr 10, 2026

Uh oh!

cardil commented Apr 17, 2026

Uh oh!

codecov bot commented Apr 17, 2026 •

edited

Loading

Uh oh!

cardil commented Apr 18, 2026

Uh oh!

cardil commented Apr 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

cardil commented Apr 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Problem

Solution

New Tools

Build tag approach

Changes

Uh oh!

knative-prow bot commented Apr 9, 2026

Uh oh!

matejvasek commented Apr 9, 2026

Uh oh!

lkingland commented Apr 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cardil commented Apr 10, 2026

Uh oh!

cardil commented Apr 17, 2026

Uh oh!

codecov bot commented Apr 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

cardil commented Apr 18, 2026

Uh oh!

cardil commented Apr 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cardil commented Apr 9, 2026 •

edited

Loading

lkingland commented Apr 10, 2026 •

edited

Loading

codecov bot commented Apr 17, 2026 •

edited

Loading